Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Survey

Building autonomous machines that can explore open-ended environments, discover possible interactions and build repertoires of skills is a general objective of artificial intelligence. Developmental approaches argue that this can only be achieved by $autotelic$ $agents$: intrinsically motivated learning agents that can learn to represent, generate, select and solve their own problems. In recent years, the convergence of developmental approaches with deep reinforcement learning (RL) methods has been leading to the emergence of a new field: $developmental$ $reinforcement$ $learning$. Developmental RL is concerned with the use of deep RL algorithms to tackle a developmental problem -- the $intrinsically$ $motivated$ $acquisition$ $of$ $open$-$ended$ $repertoires$ $of$ $skills$. The self-generation of goals requires the learning of compact goal encodings as well as their associated goal-achievement functions. This raises new challenges compared to standard RL algorithms originally designed to tackle pre-defined sets of goals using external reward signals. The present paper introduces developmental RL and proposes a computational framework based on goal-conditioned RL to tackle the intrinsically motivated skills acquisition problem. It proceeds to present a typology of the various goal representations used in the literature, before reviewing existing methods to learn to represent and prioritize goals in autonomous systems. We finally close the paper by discussing some open challenges in the quest of intrinsically motivated skills acquisition

Grounding Spatio-Temporal Language with Transformers

Language is an interface to the outside world. In order for embodied agents to use it, language must be grounded in other, sensorimotor modalities. While there is an extended literature studying how machines can learn grounded language, the topic of how to learn spatio-temporal linguistic concepts is still largely uncharted. To make progress in this direction, we here introduce a novel spatio-temporal language grounding task where the goal is to learn the meaning of spatio-temporal descriptions of behavioral traces of an embodied agent. This is achieved by training a truth function that predicts if a description matches a given history of observations. The descriptions involve time-extended predicates in past and present tense as well as spatio-temporal references to objects in the scene. To study the role of architectural biases in this task, we train several models including multimodal Transformer architectures; the latter implement different attention computations between words and objects across space and time. We test models on two classes of generalization: 1) generalization to randomly held-out sentences; 2) generalization to grammar primitives. We observe that maintaining object identity in the attention computation of our Transformers is instrumental to achieving good performance on generalization overall, and that summarizing object traces in a single token has little influence on performance. We then discuss how this opens new perspectives for language-guided autonomous embodied agents. We also release our code under open-source license as well as pretrained models and datasets to encourage the wider community to build upon and extend our work in the future.Comment: Contains main article and supplementarie

Deep Sets for Generalization in RL

This paper investigates the idea of encoding object-centered representations in the design of the reward function and policy architectures of a language-guided reinforcement learning agent. This is done using a combination of object-wise permutation invariant networks inspired from Deep Sets and gated-attention mechanisms. In a 2D procedurally-generated world where agents targeting goals in natural language navigate and interact with objects, we show that these architectures demonstrate strong generalization capacities to out-of-distribution goals. We study the generalization to varying numbers of objects at test time and further extend the object-centered architectures to goals involving relational reasoning.Comment: 15 pages, 10 figures, published as a workshop Paper at ICLR: Beyond tabula rasa in RL (BeTR-RL). arXiv admin note: substantial text overlap with arXiv:2002.0925

Contrastive Multimodal Learning for Emergence of Graphical Sensory-Motor Communication

In this paper, we investigate whether artificial agents can develop a shared language in an ecological setting where communication relies on a sensory-motor channel. To this end, we introduce the Graphical Referential Game (GREG) where a speaker must produce a graphical utterance to name a visual referent object while a listener has to select the corresponding object among distractor referents, given the delivered message. The utterances are drawing images produced using dynamical motor primitives combined with a sketching library. To tackle GREG we present CURVES: a multimodal contrastive deep learning mechanism that represents the energy (alignment) between named referents and utterances generated through gradient ascent on the learned energy landscape. We demonstrate that CURVES not only succeeds at solving the GREG but also enables agents to self-organize a language that generalizes to feature compositions never seen during training. In addition to evaluating the communication performance of our approach, we also explore the structure of the emerging language. Specifically, we show that the resulting language forms a coherent lexicon shared between agents and that basic compositional rules on the graphical productions could not explain the compositional generalization

Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Survey

Building autonomous machines that can explore open-ended environments, discover possible interactions and autonomously build repertoires of skills is a general objective of artificial intelligence. Developmental approaches argue that this can only be achieved by autonomous and intrinsically motivated learning agents that can generate, select and learn to solve their own problems. In recent years, we have seen a convergence of developmental approaches, and developmental robotics in particular, with deep reinforcement learning (RL) methods, forming the new domain of developmental machine learning. Within this new domain, we review here a set of methods where deep RL algorithms are trained to tackle the developmental robotics problem of the autonomous acquisition of open-ended repertoires of skills. Intrinsically motivated goal-conditioned RL algorithms train agents to learn to represent, generate and pursue their own goals. The self-generation of goals requires the learning of compact goal encodings as well as their associated goal-achievement functions, which results in new challenges compared to traditional RL algorithms designed to tackle pre-defined sets of goals using external reward signals. This paper proposes a typology of these methods at the intersection of deep RL and developmental approaches, surveys recent approaches and discusses future avenues

DEEP SETS FOR GENERALIZATION IN RL

This paper investigates the idea of encoding object-centered representations in the design of the reward function and policy architectures of a language-guided reinforcement learning agent. This is done using a combination of object-wise permutation invariant networks inspired from Deep Sets and gated-attention mechanisms. In a 2D procedurally-generated world where agents targeting goals in natural language navigate and interact with objects, we show that these architectures demonstrate strong generalization capacities to out-of-distribution goals. We study the generalization to varying numbers of objects at test time and further extend the object-centered architectures to goals involving relational reasoning.International audienceThis paper investigates the idea of encoding object-centered representations in the design of the reward function and policy architectures of a language-guided reinforcement learning agent. This is done using a combination of object-wise permutation invariant networks inspired from Deep Sets and gated-attention mechanisms. In a 2D procedurally-generated world where agents targeting goals in natural language navigate and interact with objects, we show that these architectures demonstrate strong generalization capacities to out-of-distribution goals. We study the generalization to varying numbers of objects at test time and further extend the object-centered architectures to goals involving relational reasoning

Agents artificiels autotelic et sociaux : formation et exploitation de conventions culturelles chez les agents artificiels autonomes incarnés

L'un des objectifs fondamentaux de l'Intelligence Artificielle (IA) est de concevoir des agents autonomes incarnés capables d'évoluer dans divers environnements, d'accomplir une multitude de tâches et d'interagir avec les humains. À cette fin, les chercheurs en IA utilisent différentes approches, avec deux méthodes principales se distinguant : la robotique développementale et les paradigmes d'IA standard. Alors que la robotique développementale modélise le développement cognitif des agents dans des environnements simplifiés, les paradigmes d'IA standard se concentrent sur les contributions algorithmiques dans des benchmarks précis et techniques. Dans cette thèse, nous prolongeons les appels récents à combler ces deux domaines et examinons le rôle des conventions culturelles dans le développement d'agents artificiels en utilisant des algorithmes d'IA de pointe.Cette recherche s'appuie sur le travail de la psychologie du développement et se concentre sur deux aspects cruciaux du développement humain, à savoir l'apprentissage autotélique et social. Le premier permet aux agents de former des répertoires de compétences ouverts en inventant et en poursuivant leurs propres objectifs, tandis que le second leur permet de communiquer, de coopérer, d'enseigner et d'organiser leurs pensées.Nos contributions sont organisées autour de deux questions scientifiques fondamentales : 1) la formation de conventions culturelles au sein de populations d'agents artificiels, et 2) l'exploitation de conventions culturelles dans leur développement cognitif.La première partie de ce manuscrit traite de la formation de conventions culturelles. Elle s'appuie sur des études récentes dans le domaine de la communication émergente pour proposer deux études computationnelles. La première étudie la formation de conventions culturelles dans le contexte écologique où les agents artificiels communiquent via un canal sensorimoteur graphique. La seconde s'inspire de la sémiotique expérimentale et étudie l'émergence de la communication dans le problème de l'architecte-constructeur : un nouveau paradigme d'apprentissage interactif où les agents ont des asymétries d'information et des affordances qui rendent l'application de l'apprentissage par renforcement multi-agent standard impossible.La seconde partie se concentre sur l'exploitation de conventions culturelles. Inspirés par les travaux pionniers de Vygotsky et d'autres psychologues, nous introduisons d'abord le framework Vygotskien d'IA autotélique. Ce framework permet aux agents d'apprentissage par renforcement d'intérioriser les interactions sociales afin de transformer leurs capacités cognitives, leur permettant de former des représentations abstraites, d'atteindre une généralisation systématique et d'explorer leur environnement de manière créative. À la suite de cette contribution conceptuelle, nous proposons deux études computationnelles. La première explore le rôle des biais inductifs dans le problème d'ancrage de langage où les agents doivent aligner leur expérience physique du monde avec les entrées linguistiques fournies par des partenaires sociaux. Notre dernière contribution computationnelle introduit l'agent IMAGINE : un agent autotélique Vygotskien qui convertit les descriptions linguistiques données par un partenaire social en objectifs atteignables. Imagine tire parti de la productivité linguistique et de la généralisation systématique pour développer de manière créative un répertoire de compétences ouvert.One of the fundamental goals of Artificial Intelligence (ai) is to design embodied autonomous agents that can evolve in various environments, perform a multitude of tasks and interact with humans. To this end, ai researchers employ various approaches, with two primary methods standing out: developmental robotics and standard ai paradigms. While developmental robotics models agents’ cognitive development in simplified environments, standard AI paradigms focus on algorithmic contributions in precise and technical benchmarks. In this thesis, we extend upon recent calls to bridge these two fields and investigate the role of cultural conventions in the development of artificial agents using state-of-the-art ai algorithms. This research leverages work from developmental psychology and focuses on two crucial aspects of human development, namely autotelic and social learning. The former enables agents to form open-ended repertoires of skills by inventing and pursuing their own goals while the latter enables them to communicate, cooperate, teach, and organize their thoughts. Our contributions are organized around two fundamental scientific questions: 1) the formation of cultural conventions within populations of artificial agents, and 2) the exploitation of cultural conventions in their cognitive development. The first part of this manuscript deals with the formation of cultural conventions. It builds on recent studies in the field of emergent communication to propose two computational studies. The first one investigates the formation of cultural conventions in the ecological context where artificial agents communicate via a graphical sensory-motor channel. The second one draws inspiration from experimental semiotics and studies the emergence of communication in the architect-builder problem: a novel interactive learning paradigm where agents have asymmetries of information and affordances which makes the application of standard Multi-Agent Reinforcement Learning impossible. The second part focuses on the exploitation of cultural conventions. Inspired by the pioneering work of Vygotsky and other psychologists we first introduce the Vygotskian Autotelic ai Framework. This framework enables Reinforcement Learning agents to internalize social interactions in order to transform their cognitive abilities enabling them to form abstract representations, achieve systematic generalization, and creatively explore their environment. Following this conceptual contribution, we propose two computational studies. The first one explores the role of inductive biases in the language grounding problem where agents need to align their physical experience of the world with linguistic inputs provided by social partners. Our final computational contribution introduces the imagine agent: a Vygotskian autotelic agent that converts linguistic descriptions given by a social partner into targetable goals. imagine leverages language productivity and systematic generalization to grow an open-ended repertoire of skills in a creative way

Agents artificiels autotelic et sociaux : formation et exploitation de conventions culturelles chez les agents artificiels autonomes incarnés

One of the fundamental goals of Artificial Intelligence (ai) is to design embodied autonomous agents that can evolve in various environments, perform a multitude of tasks and interact with humans. To this end, ai researchers employ various approaches, with two primary methods standing out: developmental robotics and standard ai paradigms. While developmental robotics models agents’ cognitive development in simplified environments, standard AI paradigms focus on algorithmic contributions in precise and technical benchmarks. In this thesis, we extend upon recent calls to bridge these two fields and investigate the role of cultural conventions in the development of artificial agents using state-of-the-art ai algorithms. This research leverages work from developmental psychology and focuses on two crucial aspects of human development, namely autotelic and social learning. The former enables agents to form open-ended repertoires of skills by inventing and pursuing their own goals while the latter enables them to communicate, cooperate, teach, and organize their thoughts. Our contributions are organized around two fundamental scientific questions: 1) the formation of cultural conventions within populations of artificial agents, and 2) the exploitation of cultural conventions in their cognitive development. The first part of this manuscript deals with the formation of cultural conventions. It builds on recent studies in the field of emergent communication to propose two computational studies. The first one investigates the formation of cultural conventions in the ecological context where artificial agents communicate via a graphical sensory-motor channel. The second one draws inspiration from experimental semiotics and studies the emergence of communication in the architect-builder problem: a novel interactive learning paradigm where agents have asymmetries of information and affordances which makes the application of standard Multi-Agent Reinforcement Learning impossible. The second part focuses on the exploitation of cultural conventions. Inspired by the pioneering work of Vygotsky and other psychologists we first introduce the Vygotskian Autotelic ai Framework. This framework enables Reinforcement Learning agents to internalize social interactions in order to transform their cognitive abilities enabling them to form abstract representations, achieve systematic generalization, and creatively explore their environment. Following this conceptual contribution, we propose two computational studies. The first one explores the role of inductive biases in the language grounding problem where agents need to align their physical experience of the world with linguistic inputs provided by social partners. Our final computational contribution introduces the imagine agent: a Vygotskian autotelic agent that converts linguistic descriptions given by a social partner into targetable goals. imagine leverages language productivity and systematic generalization to grow an open-ended repertoire of skills in a creative way.L'un des objectifs fondamentaux de l'Intelligence Artificielle (IA) est de concevoir des agents autonomes incarnés capables d'évoluer dans divers environnements, d'accomplir une multitude de tâches et d'interagir avec les humains. À cette fin, les chercheurs en IA utilisent différentes approches, avec deux méthodes principales se distinguant : la robotique développementale et les paradigmes d'IA standard. Alors que la robotique développementale modélise le développement cognitif des agents dans des environnements simplifiés, les paradigmes d'IA standard se concentrent sur les contributions algorithmiques dans des benchmarks précis et techniques. Dans cette thèse, nous prolongeons les appels récents à combler ces deux domaines et examinons le rôle des conventions culturelles dans le développement d'agents artificiels en utilisant des algorithmes d'IA de pointe.Cette recherche s'appuie sur le travail de la psychologie du développement et se concentre sur deux aspects cruciaux du développement humain, à savoir l'apprentissage autotélique et social. Le premier permet aux agents de former des répertoires de compétences ouverts en inventant et en poursuivant leurs propres objectifs, tandis que le second leur permet de communiquer, de coopérer, d'enseigner et d'organiser leurs pensées.Nos contributions sont organisées autour de deux questions scientifiques fondamentales : 1) la formation de conventions culturelles au sein de populations d'agents artificiels, et 2) l'exploitation de conventions culturelles dans leur développement cognitif.La première partie de ce manuscrit traite de la formation de conventions culturelles. Elle s'appuie sur des études récentes dans le domaine de la communication émergente pour proposer deux études computationnelles. La première étudie la formation de conventions culturelles dans le contexte écologique où les agents artificiels communiquent via un canal sensorimoteur graphique. La seconde s'inspire de la sémiotique expérimentale et étudie l'émergence de la communication dans le problème de l'architecte-constructeur : un nouveau paradigme d'apprentissage interactif où les agents ont des asymétries d'information et des affordances qui rendent l'application de l'apprentissage par renforcement multi-agent standard impossible.La seconde partie se concentre sur l'exploitation de conventions culturelles. Inspirés par les travaux pionniers de Vygotsky et d'autres psychologues, nous introduisons d'abord le framework Vygotskien d'IA autotélique. Ce framework permet aux agents d'apprentissage par renforcement d'intérioriser les interactions sociales afin de transformer leurs capacités cognitives, leur permettant de former des représentations abstraites, d'atteindre une généralisation systématique et d'explorer leur environnement de manière créative. À la suite de cette contribution conceptuelle, nous proposons deux études computationnelles. La première explore le rôle des biais inductifs dans le problème d'ancrage de langage où les agents doivent aligner leur expérience physique du monde avec les entrées linguistiques fournies par des partenaires sociaux. Notre dernière contribution computationnelle introduit l'agent IMAGINE : un agent autotélique Vygotskien qui convertit les descriptions linguistiques données par un partenaire social en objectifs atteignables. Imagine tire parti de la productivité linguistique et de la généralisation systématique pour développer de manière créative un répertoire de compétences ouvert

Agents artificiels autotelic et sociaux : formation et exploitation de conventions culturelles chez les agents artificiels autonomes incarnés

One of the fundamental goals of Artificial Intelligence (ai) is to design embodied autonomous agents that can evolve in various environments, perform a multitude of tasks and interact with humans. To this end, ai researchers employ various approaches, with two primary methods standing out: developmental robotics and standard ai paradigms. While developmental robotics models agents’ cognitive development in simplified environments, standard AI paradigms focus on algorithmic contributions in precise and technical benchmarks. In this thesis, we extend upon recent calls to bridge these two fields and investigate the role of cultural conventions in the development of artificial agents using state-of-the-art ai algorithms. This research leverages work from developmental psychology and focuses on two crucial aspects of human development, namely autotelic and social learning. The former enables agents to form open-ended repertoires of skills by inventing and pursuing their own goals while the latter enables them to communicate, cooperate, teach, and organize their thoughts. Our contributions are organized around two fundamental scientific questions: 1) the formation of cultural conventions within populations of artificial agents, and 2) the exploitation of cultural conventions in their cognitive development. The first part of this manuscript deals with the formation of cultural conventions. It builds on recent studies in the field of emergent communication to propose two computational studies. The first one investigates the formation of cultural conventions in the ecological context where artificial agents communicate via a graphical sensory-motor channel. The second one draws inspiration from experimental semiotics and studies the emergence of communication in the architect-builder problem: a novel interactive learning paradigm where agents have asymmetries of information and affordances which makes the application of standard Multi-Agent Reinforcement Learning impossible. The second part focuses on the exploitation of cultural conventions. Inspired by the pioneering work of Vygotsky and other psychologists we first introduce the Vygotskian Autotelic ai Framework. This framework enables Reinforcement Learning agents to internalize social interactions in order to transform their cognitive abilities enabling them to form abstract representations, achieve systematic generalization, and creatively explore their environment. Following this conceptual contribution, we propose two computational studies. The first one explores the role of inductive biases in the language grounding problem where agents need to align their physical experience of the world with linguistic inputs provided by social partners. Our final computational contribution introduces the imagine agent: a Vygotskian autotelic agent that converts linguistic descriptions given by a social partner into targetable goals. imagine leverages language productivity and systematic generalization to grow an open-ended repertoire of skills in a creative way.L'un des objectifs fondamentaux de l'Intelligence Artificielle (IA) est de concevoir des agents autonomes incarnés capables d'évoluer dans divers environnements, d'accomplir une multitude de tâches et d'interagir avec les humains. À cette fin, les chercheurs en IA utilisent différentes approches, avec deux méthodes principales se distinguant : la robotique développementale et les paradigmes d'IA standard. Alors que la robotique développementale modélise le développement cognitif des agents dans des environnements simplifiés, les paradigmes d'IA standard se concentrent sur les contributions algorithmiques dans des benchmarks précis et techniques. Dans cette thèse, nous prolongeons les appels récents à combler ces deux domaines et examinons le rôle des conventions culturelles dans le développement d'agents artificiels en utilisant des algorithmes d'IA de pointe.Cette recherche s'appuie sur le travail de la psychologie du développement et se concentre sur deux aspects cruciaux du développement humain, à savoir l'apprentissage autotélique et social. Le premier permet aux agents de former des répertoires de compétences ouverts en inventant et en poursuivant leurs propres objectifs, tandis que le second leur permet de communiquer, de coopérer, d'enseigner et d'organiser leurs pensées.Nos contributions sont organisées autour de deux questions scientifiques fondamentales : 1) la formation de conventions culturelles au sein de populations d'agents artificiels, et 2) l'exploitation de conventions culturelles dans leur développement cognitif.La première partie de ce manuscrit traite de la formation de conventions culturelles. Elle s'appuie sur des études récentes dans le domaine de la communication émergente pour proposer deux études computationnelles. La première étudie la formation de conventions culturelles dans le contexte écologique où les agents artificiels communiquent via un canal sensorimoteur graphique. La seconde s'inspire de la sémiotique expérimentale et étudie l'émergence de la communication dans le problème de l'architecte-constructeur : un nouveau paradigme d'apprentissage interactif où les agents ont des asymétries d'information et des affordances qui rendent l'application de l'apprentissage par renforcement multi-agent standard impossible.La seconde partie se concentre sur l'exploitation de conventions culturelles. Inspirés par les travaux pionniers de Vygotsky et d'autres psychologues, nous introduisons d'abord le framework Vygotskien d'IA autotélique. Ce framework permet aux agents d'apprentissage par renforcement d'intérioriser les interactions sociales afin de transformer leurs capacités cognitives, leur permettant de former des représentations abstraites, d'atteindre une généralisation systématique et d'explorer leur environnement de manière créative. À la suite de cette contribution conceptuelle, nous proposons deux études computationnelles. La première explore le rôle des biais inductifs dans le problème d'ancrage de langage où les agents doivent aligner leur expérience physique du monde avec les entrées linguistiques fournies par des partenaires sociaux. Notre dernière contribution computationnelle introduit l'agent IMAGINE : un agent autotélique Vygotskien qui convertit les descriptions linguistiques données par un partenaire social en objectifs atteignables. Imagine tire parti de la productivité linguistique et de la généralisation systématique pour développer de manière créative un répertoire de compétences ouvert

Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: A Short Survey

International audienceBuilding autonomous machines that can explore open-ended environments, discover possible interactions and build repertoires of skills is a general objective of artificial intelligence. Developmental approaches argue that this can only be achieved by autotelic agents: intrinsically motivated learning agents that can learn to represent, generate, select and solve their own problems. In recent years, the convergence of developmental approaches with deep reinforcement learning (rl) methods has been leading to the emergence of a new field: developmental reinforcement learning. Developmental rl is concerned with the use of deep rl algorithms to tackle a developmental problem-the intrinsically motivated acquisition of open-ended repertoires of skills. The self-generation of goals requires the learning of compact goal encodings as well as their associated goal-achievement functions. This raises new challenges compared to standard rl algorithms originally designed to tackle pre-defined sets of goals using external reward signals. The present paper introduces developmental rl and proposes a computational framework based on goal-conditioned rl to tackle the intrinsically motivated skills acquisition problem. It proceeds to present a typology of the various goal representations used in the literature, before reviewing existing methods to learn to represent and prioritize goals in autonomous systems. We finally close the paper by discussing some open challenges in the quest of intrinsically motivated skills acquisition